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ABSTRACT 

The field of education is undergoing fundamental change with the work processes. Xerox has created the Xerox Ignite™ Educator 


growing use of data. Fine-scale data collection at the item-response 
level is now possible. Xerox has developed a system that bridges the 
paper-to-digital divide by providing the well-established and easy-to- 
use paper interface to students, but digitizes the responses for 
scoring, validating, reporting, and managing data using a range of 
digital technologies. The Ignite™ system supports written responses, 
shading, connecting lines, multiple choice selections, and other 
question types. For some users, monitoring is at a very fine-grain 
level in both time and skill, while for others the data is used for more 
summative evaluations and strategic planning; one users’ details may 
be another user’s overview. All the reports presented in this 
document use the same basic atomic data elements and associated 
meta-data. The hierarchical nature of the organization of users 
requires that these atomic elements be combined in different ways 
for specialized visual representations, dependent upon the needs of 
the user. 
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1. INTRODUCTION 

Technology and regulations regarding education have increased the 
availability of student data as well as the need to track student 
performance over time. No Child Left Behind [7], Race-to-the-Top 
[9], and the Common Core State Standards [3] are all efforts within 
the United States that have endeavored to make the tracking of 
student learning growth more measurable. Despite the increase in the 
access to, and need for data and data analysis, the abilities for many 
educators to make use of available data has not kept pace with the 
need [5]. Data analysis requires knowledge and tools to which not all 
educators have easy access [13]. Enabling data to be visualized in 
ways that are familiar to educators will help encourage the use of 
student data to inform student learning and instruction. This paper 
provides one example of how reporting student data in a user 
friendly form, for many levels of users, can help educators to find 
more effective uses for their student data. 

2. BACKGROUND 

2.1 Teachers Changing Their Instruction and 
Their Needs 

Teachers have begun transitioning from curriculum-based instruction 
to student-centered instruction, which shifts the focus to assessing 
students at the beginning, middle and end of an instructional unit. In 
this way, teachers learn what the students already know about a 
particular subject from the start, where to focus needed instruction, 
and collect data throughout the process of their learning growth. 

To help support the shift to utilizing student data to inform day-to- 
day instruction in ways that fit more closely into educator’s current 


Support System [16]. Ignite™ is a web-based teacher tool for 
printing, scanning and scoring a variety of hand-marked student 
work and also manages the student data and produces personalized 
reports. Student work is generally an assessment (e.g. a quiz or test). 

The item-response level of information is defined as an atomic unit: 
“A student is presented an item on a date by a teacher and provides 
some type of response.” Each part of the atomic element contains 
additional metadata. All the reports present views of the same 
underlying data, but with differing levels of aggregation, dependent 
upon the needs of the user. This paper describes these differing user 
requirements and how a set of consistent and connected graphical 
reports can scale across the needs of these different users and their 
needs for data. 

3. RELATED WORK 

Data and data mining usage in the education domain (educational 
data mining or EDM) is relatively new. The field has grown rapidly 
for just over a decade [ 1 ] . 

There is a desire that the use of data will foster improvements at all 
levels of education. The desire for data-driven improvement in 
learning is countered by a concern that the use of data by itself will 
lead to too much of a focus on testing rather than teaching [5]. Over 
the years the focus of research has moved more into the field of 
prediction [2], and it may be that the real value will come over time 
when enough longitudinal data is available. 

Public educational institutions have a hierarchical nature. In the 
United States primary schools there is a hierarchy of superintendents, 
principals, team leaders, and classroom teachers all making 
decisions. This hierarchy of users share common tasks including the 
analysis and visualization of data, providing feedback to support 
instructors, recommendations for students, and grouping of students, 
among others. The hierarchical nature of the users within the 
educational organization presents interesting challenges in both 
EDM [2] and in the use of the data. Teachers want easy-to-use 
systems, with a “desire to see assessment results at the level of 
subscales (groups of test items) related to specific standards and at 
the level of individual items in order to tailor instructions.” [6] 
"Decisions are made at all levels of school organization. The 
superintendent makes decisions concerning a school district's goals 
and strategies. Then principals make tactical decisions concerning 
those goals and strategies to accomplish them in relation to their own 
buildings. Department heads and team leaders then make curricular 
and operational decisions to carry out the day-to-day activities of a 
department or unit. And, finally, classroom teachers make decisions 
in their classrooms”. 

Others have investigated the use of visualization in higher education 
situations with limited success [8], The use of on-line learning tools 
has led to visualizations of curriculum [4], the design of models of 
student learning [10], the use of graph structures to understand 


Proceedings of the 8th International Conference on Educational Data Mining 


468 



patterns of student enrolment [14] and the use of a small set of static 
and interactive visualizations of user data [11], 

4. USERS AND USER REQUIREMENTS 

4.1 Ethnographic Study of Pilot Deployment 

During a technical pilot of the Xerox Ignite™ application, four 
elementary schools in two school districts participated. An 
ethnographic study was conducted to observe teachers’ processes to 
identify challenges with using the pilot tool, and to collect user 
requirements and needs for the system. Observations and open-ended 
interviews [12] were the primary data collection methods to study 
teachers’ assessment practices. Many teachers expressed a desire to 
know how their students were performing in the skills being taught, 
and wanted to understand how well the students understood the 
skills. 

Talks with principals and school district administrators, including 
district data specialists, uncovered another level of requirements [29] 
relating to trend analysis, student growth over time, class-to-class 
and school-to-school comparisons, and progress monitoring. 
Principals and administrators expressed a need to see student data at 
the grade level, reaching from single classrooms, to all classrooms 
for a single grade in a single building, to entire buildings or the entire 
district. 

Just as the assessments are used for different purposes, the users of 
reports have different needs. The users have been segmented into 
three major groups: A single teacher & class, principal or lead 
teacher with several classes within a school, and a district 
administrator looking across multiple schools in the district. 

4.2 Single Teacher / Class 

A teacher working with a class, or a single student in a class, is the 
lowest level of granularity within the current scenario. In this 
situation a teacher has one of two major goals: assessing the success 
or direction of a lesson or helping a single student. 

To assess the success of a lesson plan, a teacher needs to see the 
class average, but also details about the mastery of different skills 
within that teaching unit. In Ignite™, the teacher can group and sort 
the questions on an assessment report according to the metadata 
related to the skills connected to each question. 

To help an individual student, the teacher, student, and often parent 
need fine-scale information about specific mastery of skills. The 
teacher must be able to communicate with the student and parent on 
specific problem areas that need immediate work. 

The data markers for this type of user must reveal information about 
the individual student, and the specific question or skill. Reports 
must reveal the data at the level of the basic atomic data unit, to the 
level of every item and response by a student to that item. 

4.3 Several Classes within a Grade or School 

A school principal or lead teacher within a grade or subject area 
needs a middle level of data aggregation. A principal is in charge of 
an entire school building, typically covering several grades. A lead 
teacher is typically focused on a single grade, or a single subject 
within a grade. Users at this level are typically looking at the overall 
progress of a cohort and the management of class or group 
affiliations of students. The goal is to best assign students to classes 
or groups and to insure that these classes or groups are on track to 
meet marking period and yearly goals. 


The data markers for this type of user need to reveal information 
about the class statistics and summary information about individual 
students. Reports for this level can also reveal the data so that 
cohorts can be compared, and individual outliers within classes or 
groups can be identified. Skill proficiency can be shown across time 
and across multiple classes within a given group giving a wider view 
of proficiency trends. The skills are grouped at the larger unit or 
quarterly time intervals, and not at the individual skill code level. 

4.4 Across Schools within a District 

District administrators analyse data to determine long-term trends 
and comparisons, to report to state agencies, and to evaluate 
curriculum. Users at this level are more focused on summative and 
high-stakes assessments. These users look across schools, and 
compare their own district with other districts in the area and those 
with similar social and economic demographics. 

The data markers for this class of user need to reveal information 
about the overall aggregate status of a district or school, with visual 
markers at the grade, teacher, or building level. Information about 
individual students is not identified or required. Trends for skills are 
most often limited to subject and grade level expectations. 

5. DESIGN CONSIDERATIONS 

Design choices were made in response to the user requirements 
discussed above. These design choices fell into two major areas: 
The general usability or workflow, and the visual attributes of the 
reports. 

5.1 Report Selection Workflow 

To produce a report, the user needs to select both a data set and a 
report type. These selections are not independent; as not all data can 
be rendered as any report, and each report needs an appropriate set of 
data. A linear sequential method was developed to guide the user 
through the report selection process. 

The logical concept of the selection process is shown in Figure 1. 
The first step is the top row of the selection space where the user 
specifies “Who?” i.e. the target user and student aggregation level of 
the report. The second step is the left most column of the selection 
space where the user specifies "What?” i.e. how many assessments 
are to be viewed in the report. The third step of the selection process 
defines "When?” i.e. is this report for a single instance, or does it 
cover multiple recurrences. These three linear sequential steps allow 
the user to navigate simply through a three-dimensional specification 
space ending with a choice of just a few different eligible reports. 


(l)Who? 
(2)What? (3 [When? 

Teacher / 
Class 

Principal 
or Lead 
Teacher / 
Several 
Classes 

District/ 

Multiple 

Schools 

Single 

Assessment 

Once 

• Matrix 

• Table 

• Image 

• Matrix 

• Bar 

• Distribution 

Over 

Time 

• Matrix 

• Line 

• Grouped 
Bar 

• Lines 

• Lines 






Portfolio of 
Assessments 

Once 

• Bar 

• Bar / 
Scatter 

• Distribution 

Over 

Time 

• Lines 

• Lines 



Figure 1 - Report selection table 
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The layout of the report selection table shown in Figure 1 also aids in 
understanding the automatic aggregation of data according to user 
and data selection. The different user views for the case of 
aggregating data about a single assessment given once are shown in 
Figure 2. In all cases, that same single assessment is chosen, but the 
report is different depending upon the scope of classes selected. A 
teacher meeting with a single student is most interested in the report 



Single student Entire class Several classes 


Figure 2 Flow of data aggregation and report type for a 
single assessment given once. In moving from left to right 
the question detail is lost and the higher level averages are 
presented. 



Figure 3 Flow of data aggregation and report type for a 
portfolio of four assessments. In moving from left to right 
the student specific information is lost and higher level 
distributions are presented. 

on the left; a table report that provides detailed information about 
each question. A teacher assessing the progress of an entire class is 
most interested in the report in the middle; a matrix heat map. A 
district administrator, looking at several classes is most interested in 
the report on the right; a distribution report that provides information 
about school and class averages, and some notion of the distribution 
of scores within each class and range of scores within each school. 

The different user views for the case of aggregating data about a 
portfolio of assessments are shown in Figure 3. In all cases, the same 
set of assessments is chosen, but the reports are different depending 
upon the scope of classes selected. A teacher meeting with a student 
and parent is most interested in the report on the left; a single bar 
chart report that provides a student’s performance across a range of 
skills, at a single point in time. A group of teachers meeting to 
discuss the grouping of students is most interested in the report in the 
middle; a bar chart report that provides information about the class 
average, but also the score of each student. A district administrator, 
looking at several classes is most interested in the report on the right; 
a distribution report that provides information about school and class 
averages, and some notion of the range of scores within a class. 

6. REPORT DESIGNS 

There are 6 different basic report styles: (1) image, (2) table, (3) 
matrix heat map, (4) line, (5) bar, and (6) distribution. Where 
appropriate, report styles were customizable to properly display the 
defined data set selected by the user. 

6.1 Image 

The image report is an image of the scanned and validated 
assessment. This report is the only portrait mode report, and has no 
header or footer information to maximize the actual assessment 
image region. Two examples of this report type are in Figure 4. 

The visual representation exactly matches the physical paper version 
of the assessment. 

The green highlight areas are those questions that were validated by 
the teacher as correct. Responses that were validated as incorrect 
were shown with a red overlay, skipped responses with a yellow 
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Figure 4 - Image reports where green indicates the area of a 
correct answer and red is an incorrect answer 


overlay, and those validated for partial credit were shown in blue. 
The color coding served as a pre-attentive signal to the correctness of 
the student’ s answer. 

This report is used by a teacher in individual consultations with a 
student and/or parent. It is a record of the marks the student made on 
the assessment, and how the teacher validated each question. 

6.2 Table 

The table report is a listing of the correct, partial, skipped, and 
incorrect responses to a single assessment, by a single student. The 
questions are placed in the column corresponding to the scored and 
validated response and include the question number, the question 
description (obtained from metadata), the number of points the 
student earned for the question, and the total number of points 
possible on the question. The data is presented at the fine-scale 
atomic level, with the use of metadata. By using the metadata about 
the question, the report is applicable to any type of question. If the 
question description metadata is used to encode specific skill 
information, then a quick visual scan down the list reveals common 
skills that have appeared in a single column, or the columns could be 
sorted according to some value(s) in the metadata. 

This report is used by teachers with students and parents. It is easy 
for students to understand that their goal is to make all the questions 
appear in the left-most (correct) column. 

6.3 Matrix heat map 

The matrix heat map report is a visual summary of the responses to 
each question on an assessment, by each student. The data for each 
student appears in a row, and the data for each question appears in a 
column. At each intersection point there is a graphical representation 
of the student’s response to that question. 

Sorting the rows and columns of the matrix provides the user with a 
quick visual assessment of several different stories [15]. By sorting 
the rows in order of student score, the teacher is able to quickly 
discern who has mastered the skills, and who has not as well as 
identifying which areas where most students are struggling. By 
sorting the columns in order of the correct number of student 
responses, or sorting by the metadata associated with questions, the 
teacher can quickly see what groups of questions or skill sets were 
successfully mastered by the class and which were not. If a particular 
set of question were not mastered by the class, the teacher now has 
this additional piece of information and can decide upon the need to 
re-teach a particular set of skills to that subgroup of students. 

6.4 Bar 

The Bar Chart is used in several instances. For consistency, all bar 
charts use the vertical axis to represent score. The horizontal axis can 
be categorical values of assessments, students, classes, or dates. The 


470 



bars can be singular or grouped, depending upon the amount of data 
to be displayed. 

6.4.1 Comparing Students or Classes over time 

The collection of scores for a single assessment repeated over time 
can be represented using a bar chart. In this case, the x-axis is a 
categorical list of students or classes. There is a group of bars for 
each student or class, and a bar of each instance in time. 

6.4.2 Portfolio of Assessments 

The portfolio of assessment scores for a single student, a single class, 
or all the classes within a school all use similar representation. The 
common representation provides users with a common mental model 
for scaling across such aggregations. In all these cases, the x-axis is a 
categorical list of assessments. 

For the case of a single student, there is one bar for each assessment. 

For the case of a single class, there is one bar for each class average 
score, and the distribution of student scores are overlaid as a jittered 
scatter plot, where the x-jitter is bounded to the width of the 
corresponding bar 

6.5 Line 

The line chart is used only for trends over time. For consistency, all 
line charts use the vertical axis to represent score. For a single 
assessment, the score can be absolute or percentage, and may be 
aggregated at the student, class, or school level. In all cases, the x- 
axis is a categorical list of times, e.g. the date each assessment was 
given. These reports are used to view the progress over time of one 
or more assessments. 

When viewing the portfolio of results over time, each line represents 
a different assessment. The points on the line are the relative score 
achieved on the given assessment at that point in time. The score can 
be a single student, aggregated over a class, or over a school. 

6.6 Distribution 

Reports showing a statistical summary of distributions typically are 
used only by district level users. These types of reports use the 
greatest amount of aggregation of atomic data elements. For 
consistency, all distribution charts use the vertical axis to represent 
score. The x-axis is a categorical list of assessments or schools. The 
markers are groupings of box and whisker plots. 

7. CONCLUSION 

We designed and deployed a system that implements a large and 
comprehensive set of reports for use by educators at different levels. 
The system was designed based on ethnographic studies and iterative 
participatory feedback from users, as well as subject matter experts 
on staff. The novel aspect of this work was the creation of a 
complete system that bridges the paper-digital divide, offers views 
into the data at different levels of granularity and aggregation, and 
scales to match a user’s needs and work processes while preserving 
similarity in the selection workflow and report design. 

The reports all used the same underlying fine-grain data element at 
the item-response level by each student, but aggregate the data 
differently dependent on the user’s needs. Users included teachers- 
students-parents, lead teachers or principals, and district level 
administrators. These users had needs that required scalability thru 
several levels of data aggregation. 

Report designs focused on the re-use of basic design concepts across 
the different visual representations, thus allowing users to more 
easily traverse the report space by learning a common set of patterns 
and styles. The report selection process follows a linear progression 
of selecting the collection of students, the collection of item- 
responses, the time, and finally any visual representation options for 
the data. 
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